A Survey of Optimistic Planning in Markov Decision Processes

We review a class of online planning algorithms for deterministic and stochastic optimal control problems, modeled as Markov decision processes. At each discrete time step, these algorithmsmaximize the predicted value of planning policies from the current state, and apply the first action of the best policy found. An overall recedinghorizon algorithm results, which can also be seen as a type ofmodel-predictive control. The space of planning policies is explored optimistically, focusing on areas with largest upper bounds on the value – or upper confidence bounds, in the stochastic case. The resulting optimistic planning framework integrates several types of optimism previously used in planning, optimization, and reinforcement learning, in order to obtain several intuitive algorithms with good performance guarantees. We describe in detail three recent such algorithms, outline the theoretical guarantees on their performance, and illustrate their behavior in a numerical example. Work performed in part while L. Buşoniu was with Team SequeL, INRIA Lille. He is also associated with the Automation Department, Technical University of Cluj-Napoca, Romania. A Survey of Optimistic Planning in Markov Decision Processes. By Buşoniu, Munos, and Babuška Copyright c © 2012 John Wiley & Sons, Inc. 1 2 A SURVEY OF OPTIMISTIC PLANNING IN MARKOV DECISION PROCESSES